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EXECUTIVE  SUMMARY 


At  the  suggestion  of  the  Deputy  Administrator,  a  Diagonal  Slice  Team  of 
affected  field  personnel  was  commissioned  by  the  Director  of  the  Systems 
Research  and  Development  Service  to  conduct  a  field  impact  evaluation  of 
the  Electronic  Tabular  Display  Subsystem  (ETABS) . 

The  final  result  of  the  Evaluation  Team  is  a  Field  Impact  Evaluation 
Report  showing  various  impacts,  feasible  alternatives,  advantages,  and 
disadvantages  of  these  alternatives  with  recommendations.  The  purpose 
of  these  recommendations  is  to  minimize  the  impact  of  ETABS  on  the  work 
force.  Upon  completion  of  the  Field  Impact  Evaluation  Report,  the  main 
portion  of  the  Evaluation  Team  was  dissolved. 

The  evaluation  process  was  broken  down  into  five  phases: 

Phase  I  -  Generation  of  Impact  Statements 

Phase  II  -  Verification  and  Analysis 

Phase  III  -  Quantification  of  Impact  Statements 

Phase  IV  -  Alternatives,  Recommendations,  and  Conclusions 

Phase  V  -  Disposition  of  Recommendations 

The  Evaluation  Team  met  for  1  week  of  intensive  meetings,  approximately 
every  6  weeks.  The  first  two  meetings  represented  a  learning  process 
on  group  dynamics,  and  also  about  each  others  perspectives.  The  fact 
that  we  had  done  some  learning  became  readily  apparent  at  a  subsequent 
meeting  when  we  were  able  to  break  up  into  subgroups  and  still  be 
sensitive  to  all  of  the  perspectives  reflected  by  the  entire  team. 

"Homework"  associated  with  the  ETABS  assignment  consumed  about  15  percent 
of  each  member's  time,  when  back  at  his  home  facility. 

The  team  was  lead  by  two  separately  selected  individuals.  The  "Team 
Leader,"  who  acted  as  moderator  for  the  team  also  performed  administrative 
functions . 

The  facilitator  taught  the  team  group  dynamics  and  helped  to  keep  the 
team  within  the  operating  guidelines  of  those  dynamics. 

It  is  recommended  that  a  subgroup  continue,  until  the  implementation  of 
ETABS,  to  monitor  the  recommendations  of  the  Evaluation  Team  and  to 
determine  the  success  of  the  process.  The  Evaluation  Assessor  Subgroup 
within  the  team  had  generated  methods  to  improve  the  evaluation  process 
for  similar  projects  in  the  future. 


A  field  impact  evaluation  enjoys  the  advantage  of  surfacing  deficiency 
early  enough  in  the  program  that  it  becomes  an  improvement  to  the 
engineering  model  rather  than  a  modification  to  the  production  model 
after  it  is  in  the  field.  The  savings  in  this  area  alone,  more  thar. 
compensate  for  the  expense  of  a  Field  Impact  Evaluation. 

Each  participant  is  richer  for  his  experience  as  a  member  of  a  Diagonal 
Slice  Group.  The  additional  perspectives  attained  make  the  individual 
a  more  valuable  employee  to  the  FAA. 
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INTRODUCTION 

PURPOSE .  This  report  describes  how  to  construct  an  Evaluation  Team 
and  perform  an  assessment  of  a  new  technology  or  system. 

BACKGROUND .  The  information  in  this  report  was  generated  by  the 
Electronic  Tabular  Display  Subsystem  (ETABS)  Evaluation  Team  and 
is  designed  to  assist  future  evaluation  teams.  The  report  includes 
the  rationale  of  the  Team's  response  to  the  pitfalls  they  encountered. 
This  report  has  three  chapters.  The  first  chapter  describes  how  to 
construct  an  evaluation  team,  and  the  second  chapter  describes  how 
to  perform  the  evaluation.  Chapter  three  describes  how  to  assess 
the  evaluation  process. 

SELECTION  OF  THE  EVALUATION  PROCESS.  The  evaluation  process  does  not 
duplicate  any  existing  procedure,  policy,  or  function  performed  by 
any  segment  of  the  organization.  However,  it  does  interact  with  the 
organization's  responsibility  in  the  procurement  of  the  new  system. 

This  new  concept  of  having  field  personnel  participate  during  the 
development  phase  of  a  new  system  may  be  met  with  resistance  by 
personnel  in  headquarters.  The  inter-departmental  conflict  (head¬ 
quarters  versus  field)  should  not  be  avoided.  This  energy  must  be 
channeled  to  produce  the  best  product. 

One  major  advantage  of  an  evaluation  team  over  a  committee  or  staff 
group  is  objectivity.  Since  they  are  analyzing  someone  else's  project, 
they  do  not  have  any  preconceived  ideas.  By  virtue  of  its  makeup, 
an  evaluation  team  can  surface  a  greater  number  of  viewpoints  with 
more  candor.  A  staff  group  tends  to  have  a  "staff  view."  Also,  a 
staff  group  or  committee  feels  pressures  of  hierarchy  that  do  not 
exist  with  a  diagonal  slice  group. 

Ir.  addition,  the  short,  part-time  existence  of  the  group  prevents  the 
development  of  the  "staff  view"  and  "empire  building." 

COST  OF  EVALUATION  PROCESS .  The  cost  would  vary  according  to  the 
magnitude  of  what  it  is  to  be  evaluated.  The  cost  of  the  ETABS 
Evaluation  Team  in  employee  hours,  travel,  and  per  diem  is  estimated 
at  $120,000  (1979  dollars) .  This  is  insignificant  compared  to  the 
overall  cost  of  the  project  and  the  possible  rewards  which  could 
outweigh  the  cost  many  times  over.  The  cost  of  future  evaluations 
should  be  considerably  less  since  the  initial  evaluation  was  a 
learning  experience. 
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RECOMMENDED  PROCEDURES .  This  report  has  various  recommendations  or. 
how  to  perform  the  evaluation  of  new  technology  or  systems.  These 
procedures  were  recommended  based  on  the  experience  of  the  ETABS 
Evaluation  Team.  They  are  only  suggested  procedures  and  any  future 
evaluation  team  may  deviate  from  these  recommendations  as  much  as 
required  to  accomplish  the  assigned  task. 

6.  ATMOSPHERE  OF  EVALUATION  PROCESS.  It  may  be  impossible  to  define 
the  true  atmosphere  of  the  evaluation  process. 

Since  ETABS  was  the  first  use  of  the  evaluation  process,  everything 
was  being  designed  and  tried  dynamically.  The  amount  of  frustration 
experienced  by  each  member  was  at  times  very  high.  Meetings  and 
activities  were  conducted  with  much  conflict,  tension,  and  frustrate 
In  addition  to  the  responsibility  of  evaluating  a  new  system,  being 
placed  in  a  "fish  bowl"  caused  an  increase  in  the  amount  of  pressure 
experienced. 

The  whole  evaluation  process  was  a  difficult  learning  process  for 
everyone  involved.  However,  despite  these  difficulties,  the  over¬ 
whelming  majority  of  participants  acknowledge  the  need  for  the 
process  and  their  willingness  to  participate  in  future  evaluations. 

This  report  was  generated  by  the  following  individuals:  Brad  Cook, 
Denver  Center;  Gerald  Mikuenski,  Houston  Center;  and  Ron  Seppala, 
Boston  Center. 

Point  of  contact  for  questions  or  comments  regarding  this  report  is 
Bill  Koch,  FAA  Washington,  AEM-301,  FTS  426-8794. 
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CHAPTER  1. 


STRUCTURE  AND  IMPLEMENTATION  OF  EVALUATION  TEAM 


1.0  GENERAL.  This  chapter  describes  how  to  organize,  construct,  and 
implement  an  evaluation  team. 

1.1.  INITIATING  OFFICIAL.  The  initiating  official  should  be  the  highest 
affected  manager  responsible  for  the  operation  impacted  by  the  new 
technology.  (The  ETABS  Evaluation  Team  was  initiated  by  the  Deputy 
Administrator.  If  the  evaluation  team  process  is  to  become  the 
normal  or  accepted  way  of  doing  business,  the  authority  to  initiate 
the  process  can  be  delegated  down.)  The  initiating  official  shall 
identify  the  major  objectives  and  any  boundaries  or  limitations. 

The  initiating  official  shall  select  the  moderator  and  facilitator. 


1.2  TIMING  OF  EVALUATION  PROCESS.  The  evaluation  team  should  be 

assembled  anytime  a  change  or  a  new  technology  comes  to  the  surface 
which  would  affect  a  wide  range  of  the  overall  organization.  In 
the  case  of  technology,  the  team  should  be  established  as  soon  as 
possible  after  prototype  specifications  are  generated.  The  team 
should  be  dissolved  after  generation  of  its  recommendations.  This 
should  be  prior  to  the  generation  of  the  production  specifications. 
A  subgroup  should  continue  to  exist  until  the  disposition  of 
recommendations  is  determined.  This  subgroup  would  watchdog  the 
recommendations  to  determine  the  results  of  the  evaluation  team 
process  on  the  project. 


Flexibility 
(ability  to 
change  what 
is  happening) 


Diagram  1 


System  Requirements  Specifications 
Prototype  Specifications 

Production  Specification 
Implementation  Plan 


Knowledge  (about  what  is  happening) 


The  ETABS  Evaluation  Team  stayed  in  existence  for  approximately  8 
months.  During  that  period,  only  6  weeks  of  nonconsecutive  time 
was  spent  on  team  activities.  When  back  at  their  facilities, 
members  will  spend  in  the  neighborhood  of  15  percent  of  their  time 
on  team  related  work.  The  rationale  for  having  an  approximate 
6-week  break  between  meetings  is  to  keep  team  members  in  touch 
with  the  real  world  and  test  their  progress  after  each  phase. 


3 


1.3 


OUTPUT  OF  EVALUATION  PROCESS.  The  output  of  the  evaluation  process 
is  a  Field  Impact  Evaluation  Report  with  recommendations  regardina 
a  new  technology.  This  report  indicates  both  the  positive  and 
negative  impacts  of  this  new  technology  and  makes  recommendations 
with  quantized  alternatives  which  are  intended  to  be  acceptable  to 
the  agency.  This  report  will  have  a  higher  degree  of  credibility 
than  can  be  obtained  via  any  other  method  currently  in  use. 

1.4  MODERATOR.  The  moderator  has  a  multi-function  role.  This  individual 
should  have  to  some  degree  both  technical  knowledge  and  responsibility 
for  the  project.  This  person  shapes  the  attitude  and  environment 
of  the  team  activities.  Thus,  the  moderator  is  the  designer  of 
team  functions.  The  experience  gained  as  a  member  of  the  evaluation 
team  can  be  used  as  the  basis  to  convert  that  member  to  a  moderator 
for  a  subsequent  evaluation  team.  It  would  be  undesirable  to  have 
full-time  moderators  (i.e.,  carry  over  from  team  to  team).  The 
chance  of  them  developing  fixed  perspectives  and  curtailing 
creativity  would  increase  if  they  were  permanent  moderators.  The 
following  items  indicate  the  major  functions  performed  by  the 
moderator : 

.Performs  administrative  functions  for  the  team  and  is  the 
focal  point  between  meetings. 

.Ensures  timely  completion  of  the  task  within  established 
constraints,  without  suppressing  creativity. 

.Helps  to  keep  some  of  the  more  heated  discussions  on  track 
and  also  frees  the  facilitator  to  function  unhampered. 

.Selects  the  evaluation  team's  composition  and  membership. 

.Coordinates  establishment  of  evaluation  team  with  all 
interested  parties  including  headquarters,  regional,  and 
field  personnel. 

.Ensures  all  necessary  information,  material,  and  techniques 
are  provided  to  the  team. 

.Recommends  the  elimination  of  unproductive  members  who  are 
undermining  or  interfering  with  the  accomplishment  of  the 
objectives.  The  major  responsibility  to  remove  this  element 
from  the  team  rests  with  the  team  members. 

.Encourages  an  energetic  and  productive  team. 

.Recommends  members  to  form  assessment  assessor  subgroup. 
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1.5 


FACILITATOR.  The  facilitator  has  a  multi- function  role.  The 
facilitator  teams  up  with  the  moderator  to  participate  in  the 
design  of  the  evaluation  team  functions  and  activities.  In 
addition  to  being  a  designer,  the  facilitator  is  an  expert  in  the 
area  of  information  transfer.  This  person  does  not  become  involved 
in  the  content  of  the  team's  task,  but  oversees  the  process  the 
team  is  using  and  helps  to  keep  that  process  on  track.  (The  ETABS 
Evaluation  Team  used  a  professional  facilitator  on  a  contract 
basis  which  has  several  advantages.  One  such  advantage  is  that  the 
service  is  available  on  an  as  needed  basis.  This  type  of 
facilitator  would  not  be  contaminated  by  the  way  the  administration 
has  previously  managed  new  projects.)  The  following  items  indicate 
the  major  functions  performed  by  the  facilitator  : 

.Exposes  the  team  to  various  methods  in  accomplishing  specific 
tasks.  If  team  members  have  little  or  no  experience  in  team 
action  dynamics,  the  facilitator  provides  the  basic  information. 

.Provides  the  external  force  to  steer  unproductive  conflict 
into  constructive  conflict  and  assists  the  team  in  developing 
their  annmach  in  resolving  conflict. 

.Provides  a  variety  of  techniques  which  can  be  used  to  resolve 
problems . 

.Forces  the  evaluation  team  to  identify  and  examine  the  methods 
it  uses  in  decision  making  along  with  the  quality  of  these 
decisions. 

.Participates  in  the  establishment  and  selection  of  evaluation 
team  composition. 

.Participates  in  the  selection  of  the  assessment  assessor  subgroup. 

1.6  COMPOSITION  OF  EVALUATION  TEAM.  The  key  to  the  success  of  the 
evaluation  process  is  field  personnel  participation  at  the 
appropriate  time. 

The  evaluation  team  should  be  composed  of  people  from  a  diagonal 
slice  of  the  field  organization.  A  diagonal  slice  is  a  selection 
of  personnel  representing  as  many  organizational  perspectives  as 
possible.  This  allows  for  a  highly  creative  and  productive 
atmosphere.  The  team  is  limited  in  size  to  prevent  it  from  becoming 
cumbersome.  From  5  to  12  people  seems  to  be  a  manageable  size. 

An  essential  ingredient  of  the  diagonal  slice  is  equal  status 
within  the  team  for  all  members.  Each  member  of  the  team  must 
participate.  In  this  fashion,  all  viewpoints  are  surfaced  allowing 
all  members  to  "see"  all  facets.  This  also  prevents  domination  by 
one  or  two  members  of  the  group.  Any  attempt  to  withdraw  from 
the  team  should  be  dealt  with  in  a  positive  manner. 
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During  the  Verification  and  Analysis  Phase,  the  concept  of  the 
rolling  diagonal  slice  is  implemented.  Several  team  members  are 
replaced  by  individuals  who  have  direct  responsibility  of  the 
project  being  evaluated.  These  nc  team  members  should  have 
participated  from  the  beginning  of  the  evaluation  process  as 
information  sources,  but  not  as  active  team  members.  (ETABS 
experience:  The  new  members  had  not  participated  from  the 

beginning.  The  modification  of  the  team  composition  caused  a 
great  deal  of  confusion.  It  required  some  time  for  the  new 
members  to  establish  themselves  as  part  of  the  team.)  The  rationale 
for  having  the  new  members  is  to  ensure  the  recommendations  would 
not  get  discarded  because  of  the  lack  of  ownership  or  involvement 
by  the  personnel  responsible  for  the  project.  The  replaced 
members  become  the  evaluation  assessor  subgroup  who  are  available 
as  information  sources.  This  is  in  addition  to  their  new  role. 

.7  ESTABLISHMENT  AND  SELECTION  OF  TEAM  MEMBERSHIP.  The  moderator  as 
the  focal  point  shall  request  nominations  from  as  wide  a  spectrum 
of  the  affected  areas  as  is  practical.  The  moderator  and 
facilitator  shall  select  the  team  members  from  the  available 
candidates.  Participation  on  the  evaluation  team  is  an  excellent 
opportunity  for  personal  growth.  Exposure  to  such  a  wide  range 
of  viewpoints  and  expertise  will  improve  the  individual's  overall 
performance.  In  the  selection  of  team  members,  certain  qualities 
are  desirable.  The  following  items  indicate  these  qualities  and 
some  rationale  behind  them  : 

.A  member  must  be  able  to  function  in  a  group,  be  creative, 
intelligent,  and  able  to  handle  ambiguity  and  conflict  in  a 
constructive  fashion.  Each  member  must  be  honest,  open, 
candid,  dedicated,  flexible,  and  have  the  ability  to 
participate  under  pressure. 

.Despite  the  large  variation  in  status  between  team  members 
within  the  FAA ,  each  member  has  equal  status  within  the  team. 

No  individual  member  has  the  power  or  can  control  the  team's 
activities  as  long  as  the  membership  enforces  this  internal 
quality. 

.Team  members  should  be  experts  in  their  specific  fields.  They 
should  be  the  primary  data  source,  but  additional  data  or 
feedback  can  be  obtained  from  their  technical  peer  groups.  A 
minimal  amount  of  their  time  is  used  as  a  data  collector. 

.Each  team  member  is  expected  to  use  his  expertise  to  make 
recommendations  with  alternatives.  Some  of  their  administrative 
functions  include  writing  reports,  taking  minutes,  giving  verbal 
presentations,  and  making  arrangements  for  meetings.  Member's 
knowledge  and  perspective  must  be  expressed  even  in  the  face 
of  opposition. 
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EVALUATION  ASSESSOR  SUBGROUP.  As  stated  in  paragraph  1.6, 
Composition  of  Evaluation  Team,  the  assessor  subgroup  is  formed 
from  the  rolling  diagonal  slice. 

The  evaluation  assessor  subgroup  is  charged  with  the  responsibility 
of  improving  the  evaluation  process.  Currently,  experience  is  the 
best  method  to  improve  and  develop  the  process.  There  must  be  an 
ongoing  evaluation  within  the  team  and  from  team  to  team  to  make 
the  necessary  changes  to  perfect  the  process  and  update  this  report. 
The  assessors  are  charged  with  the  following  task:  refine  the 
evaluation  process  and  provide  a  more  objective  immediate  feedback 
to  the  team. 

The  major  functions  of  this  subgroup  is  to  write  reports  and  give 
verbal  presentations  on  their  observations  of  the  evaluation  process. 

This  subgroup  is  to  generate  an  interim  report  after  the  generation 
of  the  recommendations  indicating  the  results  of  the  evaluation 
process.  Their  final  report  is  generated  after  the  disposition  of 
recommendations  is  determined.  Much  of  the  data  used  to  generate 
these  reports  is  obtained  by  the  assessors  as  observers  of  the 
evaluation  process.  The  moderator  and  facilitator  recommend  which 
members  form  this  subgroup  with  the  team's  approval. 


7 


CHAPTER  2. 


EVALUATION  PROCESS. 


2.0  GENERAL.  The  evaluation  process  with  respect  to  this  appendix  is 
the  assessment  of  new  technology  by  field  personnel.  The  impact 
on  the  workforce  by  this  new  technology  is  described  by  both 
positive  and  negative  quantized  impact  statements.  Recommendations 
with  alternatives  are  generated  which  would  be  accepted  by  the 
agency  and  minimize  the  impact  on  the  workforce. 

The  evaluation  process  have  five  phases  as  follows: 

Phase  I  -  Generation  of  Impact  Statements 

Phase  II  -  Verification  and  Analysis 

Phase  III  -  Quantification  of  Impact  Statements 

Phase  IV  -  Alternatives,  Recommendations,  and  Conclusions 

Phase  V  -  Disposition  of  Recommendations 

2.1  PHASE  I  -  GENERATION  OF  IMPACT  STATEMENTS.  The  Generation  of 
Impact  Statements  Phase  is  probably  the  most  singular  important 
phase.  It  is  comprised  of  four  distinct  areas  as  follows: 

.Development  of  Evaluation  Team 

.Technical  Briefing  of  Team 

.Generation  of  Impact  Workbook 

.Use  of  Impact  Workbook 

This  phase  does  not  begin  or  end  with  the  first  meeting.  It  can 
extend  and  overlap  subsequent  phases.  (In  the  case  of  ETABS,  this 
phase  crystallized  at  the  third  meeting  during  the  quantification 
phase.)  The  entire  objective  of  this  phase  is  to  generate  clear, 
distinct,  stand  alone,  positive,  and  negative  impact  statements. 

(In  the  case  of  ETABS,  the  preliminary  impact  statements  were 
suggestions,  recommendations,  and  alternatives  without  identifying 
the  exact  effect  on  the  workforce. 

2.1.1  DEVELOPMENT  OF  EVALUATION  TEAM.  The  development  of  this 
evaluation  team  begins  with  the  selection  of  team  members 
by  the  moderator  and  facilitator.  After  soliciting 
nominations  from  the  appropriate  organizations,  the 
candidates  are  screened  and  selected  to  ensure  a  balanced 
diagonal  slice  composed  of  the  proper  qualities. 


The  moderator  schedules  the  first  meeting  and  selects  the 
location  based  on  the  required  technical  orientation  of 
the  team.  Included  in  opening  briefings  should  be  one 
by  a  high  level  manager  indicating  his  interest/support/ 
goals  for  the  group. 

The  moderator  and  facilitator  generate  an  information 
package  and  distribute  it  to  team  members  prior  to  the 
first  meeting.  This  advance  material  should  contain  the 
prototype  specifications  and  an  executive  summary  of  the 
new  technology  being  evaluated.  In  addition,  any  appropriate 
information  on  the  evaluation  process. 

The  first  few  days  of  the  first  meeting  are  used  to  develop 
the  evaluation  team.  This  is  accomplished  by  informal 
presentations  by  the  moderator  and  facilitator  on  the 
history  and  background  of  the  project,  along  with  the 
purpose  and  goals  of  the  evaluation  team. 

The  facilitator  should  present  various  tools  and  concepts 
on  group  dynamics.  Some  of  major  items  presented  by  the 
facilitator  should  be  use  of  diagonal  slice  groups,  the 
nature  of  highly  effective  groups,  giving  and  receiving 
feedback,  methods  of  conflict  resolution  and  problem 
solving,  and  constructive  conflict  in  discussions.  Several 
team  development  exercises  can  be  employed  to  develop  the 
required  awareness  and  skills.  (The  ETABS  Evaluation  Team 
was  introduced  to  the  Nominal  Group  Technique  (NGT)  with 
an  in-depth  exercise  in  its  use  and  application.  The  team 
was  also  exposed  to  PIT  (Polemics • Inquiry -Theorizing) 
sessions  which  became  a  continuously  used  tool.) 

The  objective  of  the  team  orientation  process  is  to  integrate 
a  group  of  individuals  into  a  productive  team.  Any  method 
which  generates  a  high  trust  and  acceptance  level  within 
the  team  in  the  shortest  possible  timeframe  can  be  employed. 
(In  the  case  of  ETABS,  during  the  first  meeting,  long  hours 
and  continuous  contact  were  successful  in  diverting  energy 
away  from  keeping  up  personal  appearances  to  bringing  out 
the  true  personal  characteristics  of  the  individuals.  Once 
it  was  evident  that  the  trust  and  acceptance  were  present, 
each  individual  became  more  open,  candid,  and  productive.) 

It  is  recommended  that  all  meetings  be  held  away  from 
official  offices,  preferably  in  hotels  or  motels.  (A 
portion  of  the  second  ETABS  meeting  was  held  at  a  regional 
office.  This  period  of  time  was  not  effective.  The  rationale 
to  hold  the  meeting  at  the  regional  office  was  based  on 
the  fact  that  the  Regional  Director  was  assigned  overseer 
responsibility  of  the  ETABS  project.  It  is  suggested  that 
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one  of  the  members  of  the  evaluation  team  be  from  that 
particular  regional  office.  This  individual  could  provide 
the  Director  with  any  required  information.  In  addition, 
this  individual  should  seek  assistance  from  the  Director 
if  serious  problems  threaten  the  performance  of  the 
evaluation  process.  The  Regional  Director's  involvement 
in  the  evaluation  process  should  be  limited  to  the 
evaluation  and  disposition  of  the  recommendations.) 

The  development  and  growth  of  the  team  is  an  ongoing  process, 
which  ends  with  the  termination  of  the  team. 

2.1.2  TECHNICAL  BRIEFING  OF  TEAM.  The  technical  orientation 
begins  with  the  advance  material  and  briefing  by  a 
knowledgeable  individual.  The  amount  and  depth  of 
technical  information  presented  about  the  new  technology 
should  be  of  sufficient  proportion  to  ensure  an  under¬ 
standing  of  its  functional  capabilities  and  the  basic 
design  philosophy  behind  it.  Each  member  will  need  a 
different  level  of  information  based  upon  his  expertise 
and  perspective. 

A  demonstration  of  a  prototype,  model,  mock-up,  or  bread 
board  version  of  the  new  technology  is  extremely  beneficial. 

A  series  of  presentations  or  briefings  by  knowledgeable 
personnel  including  question  and  answer  periods  is  very 
useful.  The  team  should  be  given  the  means  to  obtain 
further  technical  information  as  the  need  arises.  The 
members  to  be  added  by  the  rolling  diagonal  slice  concept 
could  provide  this  additional  technical  knowledge  or  at 
least  the  means  to  obtain  it. 

2.1.3  GENERATION  OF  IMPACT  WORKBOOK.  The  generation  of  an  impact 
workbook  is  the  first  goal  of  the  evaluation  team.  The 
purpose  of  an  impact  workbook  is  to  provide  the  evaluation 
team  a  common  format  to  assemble  field  impact  information 
regarding  the  new  technology.  The  workbook  should  contain 
all  possible  factors  of  impact  to  be  investigated  by  the 
team.  The  data  collected  is  assembled  and  verified  in 
Phase  II,  Verification  and  Analysis. 

2.1.4  USE  OF  IMPACT  WORKBOOK.  The  individual  members  of  the  team 
are  to  take  this  workbook  back  to  their  facilities  and 
decide  if  these  preliminary  impact  factors  have  any  validity 
and  to  what  degree.  (In  the  case  of  the  ETABS  Evaluation 
Team,  the  previously  generated  technology  assessment 
workbook  was  expanded  and  adapted  as  the  basic  tool  to 
collect  data.  The  method  to  collect  this  data  was  discussed, 
but  it  was  left  up  to  each  member  to  perform  this  task  as 

it  seemed  appropriate.) 
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It  is  recommended  that  the  nominal  group  technique  be  used 
to  generate  the  impact  factors  to  be  considered.  (In  the 
case  of  the  ETABS  evaluation,  the  decision  to  use  the 
previously  generated  technology  assessment  workbook  was 
challenged  by  several  members.  Despite  the  fact  that  many 
factors  in  this  workbook  were  inappropriate,  not  one  single 
fact  or  question  was  removed  from  it  in  generating  the 
ETABS  workbook.  It  was  repeatedly  stated  by  one  of  the 
originators  of  the  technology  workbook  that  nothing  in  it 
was  sacred  but  the  ETABS  Team  still  treated  it  as  such. 

During  the  third  meeting  while  performing  the  quantification 
function,  it  became  clear  to  the  team  that  the  use  of  the 
technology  workbook  was  a  mistake.) 

Early  during  each  meeting,  one  member  of  the  team  should  be 
assigned  the  role  of  decision  tracker.  This  individual 
shall  prepare  the  minutes  for  that  meeting,  and  record  all 
decisions  and  unanswered  questions  generated  by  the  team. 

In  the  generation  of  the  workbook,  three  items  may  appear 
which  should  be  resolved  at  that  time.  (In  the  case  of 
ETABS,  the  resolution  of  some  of  these  items  was  postponed 
until  later  meetings.)  These  items  are  as  follows: 

■How  decisions  are  to  be  made.  It  is  recommended  that 
decisions  be  made  by  consensus  and  that  all  outputs 
from  the  team  be  acceptable  to  each  member.  This  does 
not  prevent  the  generation  of  any  minority  report. 

.How  to  handle  irresolvable  items.  (In  the  case  of 
ETABS,  the  majority  rule  technique  was  established  to 
handle  irresolvable  items.  Once  this  was  established 
and  used  on  an  irresolvable  item,  the  situation  never 
arose  again.) 

.How  to  cope  with  conflict.  Many  people  have  a 
tendency  to  suppress  conflict  with  people  that  they 
are  not  familiar  with.  This  suppression  can  reduce 
the  quality  of  the  final  product.  (In  the  case  of 
ETABS,  the  PIT  sessions,  with  its  role  playing, 
assisted  in  surfacing  conflict  and  acceptable  methods 
of  handling  it.  These  sessions  produced  a  high  level 
of  openness,  acceptance,  and  trust.  Several  members 
objected  to  these  sessions  and  in  reality  by  the  third 
meeting  the  PIT  sessions  had  accomplished  enough  to 
render  their  use  unnecessary.) 

After  the  generation  of  the  workbook,  the  team  should  clarify 
exactly  what  their  homework  assignment  between  meetings  is. 
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The  assignment  is  to  return  to  their  facility  and  answer 
the  questions  in  the  workbook.  As  the  experts,  they  can 
answer  these  questions  or  collect  additional  data  from  their 
peer  group.  During  this  task,  they  can  generate  additional 
items  of  impact  they  may  discover.  (In  the  case  of  ETABS 
much  confusion  was  encountered  in  performing  the  homework 
assignments.  This  was  based  on  a  lack  of  clear  understanding 
of  assignments;  despite  the  fact  that  during  each  meeting 
everyone  indicated  a  complete  understanding  of  their 
assignments.  During  the  third  meeting,  an  exercise  was 
used  to  determine  how  well  individuals  understood  their 
assignment  and  to  reinforce  their  understanding.  This 
exercise  assumed  that  they  had  completed  the  assignment  and 
then  the  data  was  applied  to  the  task  at  hand.) 

As  this  point,  the  team  should  decide  on  the  goals,  location, 
and  dates  of  the  next  meeting.  The  recommended  goals  are 
stated  in  Phase  II,  Verification  and  Analysis. 

.2  PHASE  II  -  VERIFICATION  AND  ANALYSIS.  During  the  Verification  and 
Analysis  Phase,  all  the  collected  data  on  the  various  preliminary 
impact  statements  and  questions  are  consolidated,  examined,  and 
substantiated.  The  result  of  this  phase  is  the  final  impact  state¬ 
ment  document.  This  phase  begins  by  collecting  data  (remember,  a 
team  member  can  be  the  sole  source)  and  verifying  the  preliminary 
impact  statement  at  the  team  member's  facility.  It  continues  with 
the  consolidation  of  each  member's  input  into  a  substantiated  impact 
statement  that  has  been  analyzed  and  accepted  by  the  team.  It  ends 
with  the  generation  of  the  final  impact  statement  report. 

A  majority  of  the  work  in  this  phase  is  performed  at  the  second 
evaluation  meeting.  This  meeting  has  three  major  activities. 

. Consolidation 

.Team  Modification 

.Introduction  to  Quantification 

The  previously  indicated  activities  are  performed  to  accomplish  the 
two  goals  for  the  second  meeting. 

.Generate  a  draft  impact  evaluation  document  which  describes 
the  nature  and  extent  of  all  known  impacts. 

.Develop  an  understanding  of  quantification  in  terms  of 
total  system  performance. 
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2.2.1  CONSOLIDATION .  The  consolidation  activity  combines  all  the 
various  input  data  and  comments  presented  by  team  members 
into  a  draft  impact  evaluation  report.  During  this  activity, 
the  data  is  analyzed  and  verified  by  team  members  to  ensure 
its  accuracy.  (In  the  case  of  ETABS,  over  600  individual 
comments  were  discussed  and  processed  in  the  preparation  of 
the  rough  draft  report.) 

The  moderator  and  facilitator  as  the  team's  designers  will 
prepare  a  schedule  of  the  team's  activities.  This  schedule 
should  have  enough  flexibility  in  it  to  allow  any  required 
changes.  The  flexibility  should  allow  for  the  possibility 
of  increasing  and  decreasing  the  number  of  days  for  any 
given  meeting.  The  moderator  and  facilitator  should  present 
various  methods  of  how  to  consolidate  and  evaluate  all  the 
available  data.  (In  the  case  of  ETABS,  the  moderator  and 
facilitator  forced  an  unacceptable  method  on  the  team. 

This  method  was  to  break  the  team  into  small  groups  which 
would  consolidate  and  verify  specific  factors  and  areas  of 
impact.  This  approach  caused  confusion  and  questionable 
results  and  was  abandoned  after  a  short  period  of  time. 

The  method  may  have  been  unsuccessful  because  of  the  diagonal 
slice  with  its  numerous  different  perspectives.  A  group 
discussion  was  held  on  each  and  every  factor,  comment,  and 
question.  The  information  was  analyzed,  consolidated,  and 
verified  by  the  entire  team.  This  method  took  a  great  deal 
of  time  and  the  first  few  items  received  much  more 
attention  than  the  last  items.  The  technique  of  working  in 
small  groups  was  attempted  unsuccessfully  several  times 
and  was  finally  used  successfully  during  Phase  IV, 
Recommendations.  The  reason  for  its  eventual  success  may 
have  been  an  increase  in  the  trust  level  between  members 
and  confidence  that  team  members  thoroughly  understood 
each  other's  position.) 

The  consolidation  impact  evaluation  statements  are  then 
organized  and  form  the  rough  draft  impact  evaluation  report. 

2.2.2  TEAM  MODIFICATION.  The  purpose  of  modifying  the  team's 
composition  is  to  accomplish  the  required  objectives  using 
the  most  efficient  method. 

As  many  subgroups  as  required  can  be  formed.  In  paragraph 
1.6,  Composition  of  Evaluation  Team,  the  rolling  diagonal 
slice  method  of  modifying  the  evaluation  team  is  described. 
(In  the  case  of  ETABS,  this  was  performed  to  accomplish 
the  two  assigned  objectives.)  The  addition  of  headquarters 
personnel  should  not  reduce  the  validity  of  the  field  impact 
statements  since  they  are  added  after  the  generation  of  the 
rough  draft  impact  evaluation  report. 
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During  this  activity,  the  creation  of  the  evaluation  assessor 
subgroup  is  accomplished.  The  detail  data  concerning  the 
purpose  and  responsibility  of  this  subgroup  are  identified 
in  chapter  3  of  this  document. 

It  is  envisioned  that  the  need  for  this  particular  subgroup 
would  decrease  and  be  eliminated  by  future  teams  as  the 
evaluation  process  is  otpimized.  (In  the  case  of  ETABS,  an 
edit  subgroup  was  formed  to  revise  the  rough  draft  impact 
statements  into  a  near  finished  product.  All  members  of  the 
team  were  to  provide  this  subgroup  with  any  additional  data. 
The  edit  subgroup  was  to  meet  just  prior  to  the  next  meeting. 
The  subgroup  was  to  have  complete  rewrite  license  to  revise 
all  the  statements.  The  approach  is  excellent;  however,  it 
was  unsuccessful  because  of  the  lack  of  direction  given  and 
received.  The  edit  subgroup  did  not  take  full  advantage  of 
their  complete  rewrite  license.) 

2.2.3  INTRODUCTION  TO  QUANTIFICATION.  The  purpose  of  the 

introduction  to  quantification  activity  is  to  prepare  the 
team  for  Phase  III,  Quantification  of  Impact  Statements. 

This  activity  is  intended  to  identify  all  the  possible  terms 
which  would  express  the  size  of  an  impact  statement  in 
relation  to  the  total  system  performance.  The  quantifiers 
should  be  defined  by  the  team  during  this  activity.  In  the 
case  of  ETABS  this  was  to  quantify  the  various  impacts  of 
the  evaluation  report  in  terms  of  the  following  items  : 

. Staffing 

.Workload 

.Availability 

. Reliability 

.Labor  Management  Relations 
.Manpower 

.Operational  Impact 
. Dollars 

Quantifications  were  to  be  considered  on  a  reasonable  worst 
case  basis. 

PHASE  III  -  QUANTIFICATION  OF  IMPACT  STATEMENTS.  The  purpose  of  the 
quantification  phase  is  to  take  each  impact  statement  and  determine 
how  significant  this  impact  is  in  understandable  and/or  measurable 
terms. 
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Phase  III  begins  during  the  second  meeting  with  the  introduction  to 

quantification.  The  majority  of  the  work  is  performed  prior  to  the  , 

third  meeting  in  the  form  of  homework.  Team  members  calculate  the 
size  of  the  impact  at  their  facilities.  It  is  completed  at  the 
third  meeting  where  all  the  data  is  consolidated  and  verified. 

At  the  completion  of  this  phase,  the  evaluation  team  should  have 
generated  a  document  which  contains  quantized  impact  statements. 

Some  forethought  should  be  given  to  the  format  of  this  document. 

(For  example,  in  the  case  of  ETABS,  delaying  a  decision  for  ai. 
acceptable  format  until  Phase  IV,  Recommendations,  caused  serious 
problems  in  writing  both  the  original  impact  statements  and  the 
quanti zed  statements . ) 

; 

Each  quantized  statement  that  is  generated  goes  through  a 
verification  process. 

2.3.1  QUANTIFICATION .  Each  team  member,  who  has  expertise  or 
other  sources  of  expertise  available  on  specific  impact 
statements,  generates  a  sizing  (quantification)  statement. 

In  most  cases,  the  resources  to  calculate  accurately  the 
sizing  of  these  impacts  is  limited  to  the  human  resources 
within  the  team. 

These  sizing  statements  are  expressed  in  the  terms 
generated  during  the  introduction  to  quantification 
activity  in  Phase  II. 

Since  much  of  this  data  is  generated  between  meetings, 
this  permits  investigation  and  analysis  of  the  sizing 
statements . 

2.3.2  VERIFICATION.  The  purpose  of  the  verification  process 
is  to  ensure  that  the  sizing  statements  are  as  accurate 
and  realistic  as  possible. 

This  is  performed  using  the  complete  resources  and  expertise 
of  the  entire  team.  Each  quantized  statement  is  tested 
and  analyzed  in  a  impartial  and  honest  manner. 

PHASE  IV  -  ALTERNATIVES,  RECOMMENDATIONS,  AND  CONCLUSIONS.  The 
purpose  of  this  phase  is  to  select  from  feasible  alternatives  the 
best  recommendations  to  reduce  and/or  eliminate  unacceptable  impacts 
generated  by  the  new  technology. 

The  phase  begins  after  the  completion  of  the  third  meeting  as  an 
assigned  homework  task.  Each  team  member  is  to  generate  feasible 
alternatives  with  cost  estimates  where  possible,  to  reduce  and/or 
eliminate  undesirable  impacts. 
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The  phase  is  completed  with  the  selection  of  the  most  realistic 
alternatives  to  make  the  new  technology  as  acceptable  as  possible 
to  the  workforce. 

There  are  three  parts  to  Phase  IV  as  follows: 

.Generation  and  consolidation  of  alternatives 

.Feasibility 

.Selection  of  alternatives 

The  generation  of  the  recommendation  is  only  second  in  its  importance 
to  the  generation  of  the  impact  statements.  This  phase  completes  a 
majority  of  the  work  of  the  evaluation  team  and  produces  the  final 
product,  i.e.,  how  to  resolve  the  problems  of  impact. 

2.4.1  GENERATION  AND  CONSOLIDATION  OF  ALTERNATIVES.  Prior  to 
the  fourth  meeting,  each  member  prepares  a  list  of 
alternatives  to  resolve  the  problems  identified  by  the 
quantized  impact  statements.  Then  they  generate  the  best 
cost/feasibility  data  available. 

During  the  fourth  meeting,  the  alternatives  are  consolidated 
and  clarified. 

This  activity  is  by  far  the  most  creative  and  innovative  in 
the  entire  evaluation  process.  Members  using  their  expertise 
and  understanding  as  to  what  field  personnel  really  need,  and 
have  an  opportunity  to  propose  new  ways  in  which  to  do 
business . 

2.4.2  FEASIBILITY.  The  purpose  of  the  feasibility  activity  is  to 
analyze  and  verify  suggested  alternative  solutions  to 
undesirable  impacts.  During  this  activity,  team  members  will 
start  to  put  forth  a  determined  effort  for  acceptance  of 
alternatives  which  will  be  of  greatest  advantage  to  their 
particular  specialty.  (In  the  case  of  ETABS,  Air  Traffic 
Control  and  Airway  Facilities  personnel  took  extremely 
strong  opposite  stands  on  the  feasibility  of  certain 
alternatives . ) 

All  alternatives  which  are  considered  to  be  unfeasible  are 
discarded.  The  determination  of  feasibility  is  reached  by 
team  consensus  after  completely  understanding  the  alternative. 

2.4.3  SELECTION  OF  ALTERNATIVES.  At  this  point  the  problems  have 
been  clearly  identified  and  all  feasible  solutions  have  been 
verified.  The  next  step  is  to  recommend  the  best  alternatives. 
During  this  activity  all  the  special  interest  within  the  team 
will  again  attempt  to  sway  the  team  in  making  recommendations 


16 


most  advantageous  to  them.  The  possibility  exists  for 
compromise  between  various  factors  within  the  team  in 
making  the  final  recommendations  or  for  dual 
recommendations.  (In  the  case  of  ETABS,  the  selection  of 
the  final  recommendations  was  made  by  consensus.  Much 
discussion  and  testing  of  the  different  perspectives 
occurred  prior  to  final  selections.) 

The  final  step  is  to  establish  the  priority  of  various 
recommendations.  From  the  field's  perspective,  all  the 
recommendations  should  be  implemented.  Some  recommendations 
will  have  a  more  substantial  positive  impact  to  field 
personnel  as  compared  to  others  and  should  be  given  more 
emphasis . 

This  concluded  the  evaluation  process  except  for  Phase  V, 
Disposition  of  Recommendations. 

PHASE  V  -  DISPOSITION  OF  RECOMMENDATIONS.  If  the  evaluation 
process  was  to  end  with  Phase  IV,  two  very  important  questions 
would  remain  unanswered. 

1.  Was  the  evaluation  of  the  new  technology  successful? 

2.  What  action  has  to  be  taken  to  improve  the  process? 

The  purpose  of  Phase  V  (Disposition  of  Recommendations)  is  to 
answer  these  questions.  The  following  is  purely  theoretical  since 
the  ETABS  evaluation  process  is  only  in  Phase  IV,  Alternatives, 
Recommendations,  and  Conclusions. 

To  ascertain  the  value  of  the  evaluation  process,  the  results  must 
be  evaluated  from  an  objective  viewpoint.  It  must  be  determined 
if  the  process  has  any  merit  or  if  it  is  a  waste  of  time  and  energy. 
Regardless  of  the  results  of  any  individual  evaluation,  some  action 
should  be  taken  to  improve  the  process.  To  accomplish  these  goals, 
two  activities  are  required. 

.Monitor  Results 

.Improve  Evaluation  Process 

To  perform  these  activities,  a  special  subgroup  is  formed  whose  chief 
purpose  is  objectivity.  This  subgroup  should  exist  until  the 
implementation  of  the  new  technology.  This  subgroup  will  spend  very 
little  time  in  performing  this  important  task. 
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2.5.1  MONITOR  RESULTS.  During  this  activity,  the  subgroup  shall 
determine  which  recommendations  were  implemented.  In 
addition,  they  shall  determine  if  the  recommendations  were 
worthwhile.  The  subgroup  will  investigate  and  trace  exactly 
what  happens  to  each  recommendation.  The  subgroup  will  also 
determine  if  the  predicted  quantized  impact  statements  were 
valid.  The  results  of  this  activity  will  indicate  if  the 
evaluation  process  was  successful.  It  may  also  verify  that 
the  normal  way  of  doing  business  is  also  valid. 

This  information  should  be  documented  by  generating  a 
supplement  to  the  field  impact  evaluation  of  the  new  technology. 

2.5.2  IMPROVE  EVALUATION  PROCESS.  The  purpose  of  this  activity  is 
to  improve  the  evaluation  process  by  a  thorough  critique  of 
the  process  after  the  fact.  This  Monday  morning  quarter- 
backing  activity  should  generate  a  supplement  to  this 
appendix.  This  supplement  should  indicate  pitfalls  to 

avoid  and  successful  procedures  to  use.  In  addition,  it  should 
indicate  the  emotional  climate  of  the  evaluation  process. 
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CHAPTER  3. 


ASSESSMENT  OF  FIELD  IMPACT  EVALUATION  PROCESS 


3.0  GENERAL .  Ths  purpose  of  the  Field  Impact  Evaluation  (FIE)  Team 
was  to  identify  the  impact  that  the  Electronic  Tabular  Display 
Subsystem  (ETABS)  will  have  on  the  workforce.  The  Evaluation 
Assessor  (EA)  Subgroup  is  charged  with  the  mission  of  overseeing 
the  functions  of  the  FIE  Team  and  recommending  changes  to  the 
evaluation  process. 

This  chapter  is  a  guideline  for  establishing  an  evaluation 
assessor  subgroup  and  assessing  the  field  impact  evaluation  process. 

3.1  BACKGROUND .  The  EA  Subgroup  associated  with  ETABS  was  composed  of 
three  members  of  the  Field  Impact  Evaluation  Team.  The  EA  Sub¬ 
group  is  commissioned  to  recommend  procedures  and  further  develop¬ 
ment  of  the  FIE  process.  Historically,  the  FAA  has  suffered  from 
poor  field  acceptance  when  implementing  new  systems.  The  Field 
Impact  Evaluation  procedure  is  a  realistic  method  of  addressing 
field  consideration  in  a  systematic  manner. 

The  potential  for  enhancing  field  acceptance  of  new  technology 
through  its  use  looks  very  promising. 

3.2  EVALUATION  ASSESSOR  SUBGROUP  OBJECTIVES.  Once  the  Field  Impact 
Evaluation  Project  Initiating  Official  has  determined  that  an 
ongoing  assessment  within  the  evaluation  team  is  required,  the 

EA  Subgroup  objectives  should  be  well  defined.  EA  subgroup  reports 
and  presentations  should  be  clearly  specified  when  orienting 
the  EA  Subgroup  Members.  Time  tables  and  resource  limitations 
should  be  included. 

3.3  EVALUATION  ASSESSOR  SUBGROUP  ASSEMBLY.  The  size  of  the  EA  Subgroup 
may  vary  proportionally  to  the  size  of  the  Field  Impact  Evaluation 
Team.  For  example,  the  ETABS  EA  Subgroup  consisted  of  3  of  the  13 
members.  The  subgroup  was  formed  using  the  rolling  diagonal  slice 
concept.  A  variety  of  procedures  may  be  employed  to  determine  who 
should  serve  as  subgroup  members.  It  is  suggested  that  a  list  of 
qualification  requirements,  personal  attributes,  and  characteristics 
be  established  to  aid  in  identifying  potential  candidates. 

.A  high  level  of  commitment  to  the  project 

.Writing  skills 

.Good  oral  communication  skills  (public  speaking) 

.Time  availability 
.High  conceptual  ability 
.Experienced  in  field  operations 

.Have  a  high  tolerance  level  for  frustration  and  ambiguity 
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Participation  by  the  Field  Impact  Evaluation  Team  is  considered 
essential  in  selecting  the  subgroup  membership.  In  the  case  of 
ETABS,  the  moderator  and  facilitator  nominated  individuals  but 
selection  was  by  team  consensus.  There  was  no  specific  leader 
assigned  or  established  in  the  subgroup,  but  general  guidance  was 
provided  by  the  team  facilitator. 

The  timeliness  of  the  selection  of  the  subgroup  membership  and  the 
initiation  of  their  activities  is  critical  to  the  effective  outcome 
of  the  assessment  project. 

The  subgroup  membership  selection  must  be  made  after  all  evaluation 
team  members  have  gained  the  fundamental  experience  of  the  subject 
evaluation  process.  It  must  not  occur  so  early  as  to  distract 
interest  or  weaken  their  commitment  to  the  project. 

3.4  EVALUATION  ASSESSOR  SUBGROUP  ASSESSMENT  FUNCTIONS.  The  following 
list  of  functions  were  developed  and  employed  by  the  ETABS  Assessor 
Subgroup.  It  is  presented  as  a  guideline.  It  may  not  be  the  best 
br  most  complete  list,  but  is  is  a  starting  point  for  future 
subgroups . 

.Participation  in  the  field  evaluation  process 

.Observation  of  the  methods  employed  by  field  evaluation  team 

.Examination  and  validation  testing  of  the  FIE  Team  report 
and  its  supporting  data 

. Interview  FIE  Team  members 

.Generation  of  questionnaires  to  collect  data  from  FIE  Team  members 

.Provide  constructive  feedback  to  the  evaluation  team  and  raise 
questions  pertaining  to  the  validity  of  the  process  including 
its  conclusions  and  recommendations 

. Initiate  action  to  ensure  a  procedure  be  established  to  track 
the  recommendations  generated  by  the  evaluation  team 

3.5  EVALUATION  ASSESSOR  SUBGROUP  PROCEDURES.  The  following  list  of 
initial  and  ongoing  procedures  served  the  purposes  of  the  ETABS 
Team  well.  These  procedures  can  be  tailored  to  meet  the  needs 
of  any  evaluation  assessment. 

3.5.1  INITIAL  ACTIVITIES.  The  Evaluation  Assessor  Subgroup  must 
generate  an  initial  assessment  plan  which  should  include 
the  following  activities. 
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•Determine  if  final  assessor  reports  are  required  and  if 
so,  when  and  in  what  form 

.Prepare  oral  presentations 

•Determine  a  method  of  data  verification 

•Determine  and  assign  assessor  functional  workloads 

•Develop  a  procedure  to  maintain  the  interface  with  the 
evaluation  team 

•Establish  an  interim  coordination  and  communication 
procedure  to  assist  each  other  with  homework  assignments 

3.5.2  ONGOING  EVALUATION.  Critique  of  the  ongoing  Field  Impact 
Evaluation  sessions  is  an  essential  activity  of  the  assessor 
subgroup . 

The  following  items  illustrate  the  ongoing  evaluation  process: 

•Meeting  separately  from  the  FIE  Team,  the  subgroup 
critiques  the  team  progress.  Including  the  value  of 
each  session  or  meeting. 

•Feedback  is  given  to  the  ETABS  Evaluation  Team  after  a 
review  and  analysis  of  a  completed  task. 

•Recommendations  are  made  to  improve  the  evaluation 
process. 

3.5.3  DATA  COLLECTION.  Early  implementation  of  procedures  to 
gather  and  preserve  a  wide  variety  of  data  is  considered  a 
must  activity  for  the  assessor  subgroup.  The  suggested 
methods  are  listed  below: 

•Arrange  for  the  subgroup  to  have  copies  of  all  reports 
and  working  documents  of  the  FIE  Team. 

•The  subgroup  members  should  make  a  complete  set  of 
notes  to  ensure  group  memory.  The  notes  should  capture 
the  mood,  feelings,  commitments,  and  productivity 
levels  of  the  FIE  Team. 

•Use  questionnaires  designed  to  capture  specific  data. 

This  should  provide  statistical  information  that  may  be 
useful  in  formulating  future  decisions. 

.Confidential  written  and  oral  critiques  about  the 
process  with  individual  team  members. 
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3.5.4  PRESENTATIONS .  The  Evaluation  Assessor  Subgroup  is  tasked 
to  provide  various  presentations.  These  presentations  will 
be  concerned  with  both  the  evaluation  of  new  technology 
and  the  evaluation  process.  The  following  items  indicate 
the  required  activities  to  achieve  this: 

.Correlate  all  appropriate  data  in  final  written  report 

.Develop  the  required  visual  and  oral  presentation 
package 

.Prepare  guidelines  and  changes  for  use  by  future 
evaluation  teams  as  required 


CHAPTER  4. 


ETABS  FIELD  IMPACT  EVALUATION  OBSERVATIONS 


4.0  The  original  concept  that  selection  of  ETABS  Evaluation  Team 

Members  was  a  critical  phase  of  the  process  has  proven  to  be  well 
founded.  The  feelings  and  emotions  of  team  members  ran  very  high 
during  most  meeting  sessions.  The  candor  required  by  members  of 
the  ETABS  Evaluation  Team  to  arrive  at  consensus  on  high  interest 
issues  forced  all  members  near  their  limits  of  patience  on  numerous 
occasions.  Different  team  members  threatened  to  leave  the  team  at 
times  when  progress  seemed  impossible.  On  some  occasions,  members 
withdrew  physically  and  mentally  for  short  periods  of  time,  only  to 
rejoin  the  team  of  their  own  initiative  or  to  be  drawn  back  by  other 
team  members  and  then  to  make  greater  contributions  than  before 
toward  the  resolution  of  the  current  issue.  Emotional  reactions  to 
some  issues  resulted  in  agreement  to  delay  some  discussions  and 
decisions  until  a  later  time  when  members  could  be  more  objective. 

The  selection  criteria  served  well  in  this  case  since  the  team  was 
never  dominated  by  any  single  member  and  peer  pressure  control  was 
never  necessary,  but  the  importance  of  membership  selection  is  to 
be  highly  emphasized. 

Likewise,  great  care  must  be  exercised  when  selecting  team  members 
to  serve  as  evaluation  assessors.  Additional  workload  and  stress 
of  the  dual  role  can  easily  exceed  the  commitment  level  of  some  team 
members . 

The  assessor  subgroup  functioned  primarily  as  a  leaderless  group, 
which  appears  to  have  been  the  correct  procedure  for  this  group.  We 
recommend  it  be  employed  by  future  teams. 

The  assessment  procedure  included  keeping  data  records  of  the 
evaluation  team  activities,  observing  the  progress  of  each  team 
session  while  making  notes  of  the  depth  of  concern,  feelings,  logic 
used,  and  data  presented  in  support  of  each  impact  issue.  The 
assessors  attempted  to  identify  the  pitfalls  and  traps  that 
triggered  the  highly  emotional  responses  of  team  members.  Personal 
attacks,  polarization  by  technical  specialties,  lack  of  expertise, 
unexpected  boundary  conditions,  and  inattentiveness  of  some  members 
always  resulted  in  a  loss  of  productivity.  The  assessors  observed 
in  contrast,  that  when  the  task  at  hand  was  well  defined  and 
understood,  with  adequate  data,  expertise,  and  harmony,  the  quantity 
and  quality  of  the  output  was  greatly  enhanced. 

Decision  making  by  consensus  was  agreed  to  by  all  ETABS  Field 
Evaluation  Team  Members  with  majority  vote  to  be  employed  only  when 
consensus  could  not  be  reached.  Majority  vote  was  used  only  twice 
in  5  weeks  of  meetings  of  the  ETABS  Field  Evaluation  Team.  Any  team 
member  could  halt  the  action  on  any  issue  presented  when  strong 
feelings  were  expressed  and/or  convincing  supporting  data  was  provided. 
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All  team  members  agreed:  their  experience  while  participating  in 
this  evaluation  would  benefit  the  FAA  by  making  them  more  capable 
and  valuable  employees;  the  field  acceptance  of  ETABS  would  be 
greatly  enhanced  by  the  improvements  to  the  systems  resulting  from 
the  Field  Impact  Evaluation  Team  recommendations;  that  field 
acceptance  would  also  be  good  as  a  result  of  the  credibility  gained 
by  involving  field  employees  in  addressing  field  impact  problems  in 
the  early  stages  of  systems  design. 

All  ETABS  Field  Impact  Evaluation  Team  Members  expressed  feelings  of 
confusion  and  frustration  when  directions  were  unclear  or  boundary 
conditions  were  not  well  defined.  It  was  necessary  to  struggle  with 
the  team  task  definition  at  the  beginning  of  each  series  of  meetings 
following  a  5-  or  6-week  break  between  meetings.  In  some  cases,  new 
data  or  points  of  view  gained  between  meetings  required  discussion 
and  evaluation  before  progress  could  proceed  on  the  subject  task  at 
hand.  However,  trying  some  sessions  seemed,  all  members  of  the  team 
said  they  would  serve  on  other  evaluation  teams  if  requested. 

Outbursts  of  emotional  frustrations  were  not  limited  to  field  team 
members.  The  team  facilitator  became  involved  on  two  occasions 
when  progress  lagged  and  team  members  felt  they  were  being  manipulated 
or  some  guidance  was  lacking  or  misleading  and  some  lack  of  trust 
resulted  and  the  team  facilitator  and  moderator  had  to  prove  them¬ 
selves  and  regain  the  confidence  of  the  team  members.  As  observed 
during  this  Field  Impact  Evaluation  Process,  the  facilitator  role 
was  a  very  demanding  one  because  the  team  members,  at  times,  were 
very  goal  oriented  and  impatient  to  proceed  with  the  project  at  the 
risk  of  sacrificing  performance  quality.  However,  the  team  facili¬ 
tator's  keen  observations  and  rational  logic  persisted  and  resulted 
in  a  strong  project  commitment  by  all  team  members  as  was  the  complete 
commitment  in  evidence  by  the  team  facilitator  and  moderator.  The 
evaluation  assessor  group  rated  the  overall  facilitator  performance 
as  excellent  in  the  ETABS  Field  Impact  Evaluation  Effort. 

The  moderator  role  appeared  equally  as  difficult  as  that  of  the 
facilitator  during  the  ETABS  Field  Impact  Evaluation.  Observations 
revealed  that  great  expectations  for  suggestions  and  new  techniques 
to  progress  were  directed  toward  the  moderator  by  team  members  when 
team  activities  were  bogged  down,  and  in  most  cases,  problem  summaries, 
objective  redefinition,  and  critique  were  just  the  ticket  for  renewed 
emphasis  and  productivity. 

Role  playing  was  employed  as  a  critique  technique  in  a  process 
referred  to  as  PIT  sessions  following  each  ETABS  session  for  the  first 
three  meetings  of  the  team.  Two  team  members  were  selected  to  present 
and  discuss  the  Pro's  of  each  session  and  likewise,  two  would  present 
and  discuss  the  Con's  of  each  session  while  acting  a  role.  This 
technique  provided  the  acceptable  cover,  enabling  open  discussion  of 
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the  sensitive  areas  limiting  progress  in  the  group  and  complementing 
the  productive  contributions  of  specific  team  members.  The  group 
tired  of  this  process  and  initiated  a  modified  version  of  PIT  without 
the  role-playing  procedure  for  the  last  two  series  of  meetings.  It 
was  felt  that  sufficient  trust  existed  among  team  members  to  enable 
open  and  candid  discussions  of  the  most  positive  and  negative  aspects 
of  team  performance  at  this  point  in  time.  Therefore,  it  is  agreed 
the  modified  PIT  was  a  successful  form  of  critique  since  the 
confidence  level  was  sufficiently  high. 

Most  ETABS  Field  Impact  Evaluation  Team  Members  expressed  concern 
that  removal  of  the  three  evaluation  assessors  from  the  mainstream 
of  activity  during  the  second  meeting  to  initiate  the  assessment 
functions  resulted  in  an  adverse  effect  on  the  team  progress  due  to 
this  loss  of  experience  and  data  resource.  They  suggested  that 
specific  assessors  be  assigned  to  conduct  the  assessor  functions  if 
assessment  is  considered  essential  and  leave  all  original  evaluation 
team  members  in  the  team.  Likewise,  it  was  recommended  that  head¬ 
quarters  and  technical  resource  members  participate  as  full  team 
members  from  the  first  meeting. 

Field  testing  of  some  of  the  ETABS  concepts  and  how  ETABS  would  be 
accepted  in  the  field  was  less  than  encouraging  since  the  system 
procurement  and  installation  is  so  far  in  the  future.  Most  field 
employees  interviewed,  concerning  the  field  impact  of  ETABS,  lost 
interest  in  discussing  the  system's  merits  and  impacts  when  informed 
that  systems  installation  would  be  some  time  in  1984  or  1985. 

The  Field  Impact  Evaluation  Team  Members  endorse  the  evaluation  process 
as  a  useful  tool  with  many  applications  in  FAA. 


